Device for increasing the dynamic range of a camera

ABSTRACT

A device for increasing the dynamic range of a camera comprising a lens (2) which forms a picture (6) of a scene (1) on a primary electronic image sensor (3) having a first dynamic range, a second image data source (4) corresponding to one picture of said scene (1), and an electronic image processing circuit (5), which combines the data from the primary image sensor (3) and the data from the second image data source (4) and provides image data representing the scene (1) as observed and represented by said image data, with a ratio between the brightest and dimmest areas of said scene (1) which is greater than the dynamic range of a primary image sensor (3). In various embodiments, the second image data source (4) is either a secondary image sensor or a memory which stores a previously sensed image from the primary image sensor (3). Said device preferably comprises a device (10) for controlling the sensitivity of the primary image sensor (3), and a transmissive, passive flat matrix screen (11) optically placed before the primary image sensor (3) to dim the brightest light rays passing therethrough.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention concerns a device for increasing the dynamic rangeof a camera so that it is tolerant of large differences in brightnessbetween the points of the scene of which an image is viewed.

2. Discussion of the Background

Its applications are mainly in shooting under uncontrolled lightconditions outdoors, in endoscopy and in welding robots.

Known tube or CCD type cameras are very intolerant of smearing andblooming and have a very low dynamic range (ratio of brightest to leastbright areas viewed correctly and simultaneously) in the order of 100.

They are also insufficiently sensitive for shooting at night.

Other devices have an image sensor and one or more passive screensdisposed optically in front of the image sensor and controlled by itwhich have a greater dynamic range but provide a highly unstable image.They provide closed loop matrix optical control and suffer from eitheroscillation of the output signal and the image transmitted or a responsetime very much greater than that of the image sensor.

SUMMARY OF THE INVENTION

The present invention intends to remedy these drawbacks by producing astable image and a wide dynamic range, possibly exceeding 100 000. Also,the image produced may embody very high sensitivity.

To achieve this the device in accordance with the present invention is adevice for increasing the dynamic range of a camera comprising anelectronic primary image sensor operating in a spectral band andsupplying image data for a range of brightness values impinging on itwith a first dynamic range, said dynamic range representing the ratio ofthe greatest brightness value to the least brightness value of thisrange, and a lens forming an image of a scene on said primary imagesensor characterized in that it comprises a second image data source,said image data coming from at least one image of said scene shotthrough said lens in the same spectral band as that of said primaryimage sensor and an image processing electronic circuit combining thedata from the primary image sensor and the data from the second imagedata source and supplying image data representing the scene viewed witha ratio of the greatest brightness value to the least brightness valueof areas of said scene represented by said output image data, said ratiobeing greater than the dynamic range of the primary image sensor.

The second image data source may comprise a secondary image sensorsensing the same image as the primary image sensor but with a differentsensitivity.

The second image data source may also comprise an image store. Imagedata stored in said image store may represent an image sensed previouslyby the primary image sensor, an image resulting from processing an imagepreviously sensed by the primary image sensor, or an image obtained froma plurality of views taken previously by the primary image sensor,combined or added together.

Other electronic circuits and additional functions are described. Inparticular, various image data processing systems can be adapted to suitthese various embodiments. They are concerned in particular withcontrolling the sensitivity of the primary image sensor over all of itssurface or in areas thereof by means of a matrix flat screen disposedoptically in front of the primary image sensor. Other additional systemsconcern the output of a signal representing an image having a monotonousresponse (in the mathematical sense) according to the brightness ofpoints in the scene viewed by the primary image sensor.

The following description given with regard to the appended drawings byway of non-limiting example only will provide a better understanding ofthe advantages, objects and features of the invention.

The dynamic range of an image sensor on which light rays impinge is theratio of the highest to the lowest brightness areas viewed correctly andsimultaneously. Between these two values the sensor outputs image datarepresenting the brightness. The lowest value is equal to thesensitivity. Above the highest value the sensor is saturated and theimage data is constant.

This description refers only to CCD type image sensors and polarizertype flat screens. Any other type of image sensor (semiconductor ortube, for example) and any other type of flat screen are within thescope of the invention, however.

The sensor comprises a photosensitive surface and circuits controllingthe operation of each point on the photosensitive surface. To clarifythe description the control circuits are not shown in the figures ordescribed in detail because they are known in themselves.

Throughout the description the word "image" denotes also the "frames"output by certain image sensors in which two frames constitute oneimage.

To clarify the description the clock signals and the power suppliesrequired by the various embodiments of the device are not shown.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic representation of a device in accordance withthe present invention.

FIG. 2 is a diagrammatic representation of a preferred first embodimentof a device in accordance with the invention comprising a single imagesensor.

FIG. 3 is a diagrammatic representation of a second embodiment of adevice in accordance with the invention comprising two image sensorswith different sensitivities.

FIG. 4 is a representation of the grid control voltages of a CCD imagesensor incorporated into the two embodiments of a device in accordancewith the invention as shown in FIGS. 2 and 3.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 shows a scene 1 and on an optical axis A and in a camera casing Bwhich is shown in part only, a flat screen 13, a lens 2 forming an image6 of the scene 1 on an electronic primary image sensor 3 and a matrixflat screen 11 disposed optically in front of the primary image sensor3. The primary image sensor 3 comprises an electronic shutter 47. Offthe optical axis and in the camera casing B are shown a second imagedata source 4, an image processing electronic circuit 5 electricallyconnected (7) to the primary image sensor, an electrical connection 8 tothe second image data source 4 and an output electrical connection 9 toa conversion table 12 which has an output electrical connection 14. Thecircuit 10 controlling the sensitivity of the primary image sensor 3 isconnected by an electrical connection 16 to the image processing circuit5, by an electrical connection 17 to the primary image sensor 3, by anelectrical connection 18 to the flat screen 13 and by an electricalconnection 15 to the matrix flat screen 11.

The electrical connections may comprise one or more electrical lines totransmit data serially or in parallel.

The lens 2 is of a known type, in particular a type known in themanufacture of video cameras, and is either interchangeable orpermanently fixed to the camera casing B.

The primary image sensor 3 is of a known type, in particular a typeknown in the manufacture of electronic and video cameras. It sensescorrectly and simultaneously brightness values in a given spectral bandin a limited range. Its sensitivity defines the lowest value in thisrange and depends on the transparency of the optical components betweenit and the scene 1 (this value is also called the aperture). Its dynamicrange is defined, for light rays impinging on the primary image sensor3, as the ratio of the brightness of the brightest to least bright areasthat it views simultaneously and correctly. Between these two extremevalues it outputs image data representing the brightness. The lowestvalue is equal to the sensitivity. Above the highest value the sensor issaturated and the image data is constant. In the remainder of thisdescription the expression "response range" refers to the range ofvalues of brightness of areas of the scene 1 for which the values ofbrightness at the primary image sensor 3 are between these two extremevalues.

The primary image sensor 3 comprises a photosensitive surface and acircuit controlling the operation of each point of the photosensitivesurface.

On the electrical connection 7 the primary image sensor 3 supplies datarepresenting the brightness received at each point of the photosensitivesurface from an area of the scene 1 and forming an image 6 on thephotosensitive surface. This image data is supplied sequentially,meaning that it represents the brightness impinging at each successivepoint of the primary image sensor 3.

The second image data source 4 provides on the electrical connection 8and simultaneously with the primary image sensor 3 image datarepresenting the scene 1 shot through the lens 2. In other words, foreach point of the scene 1 for which the lens 2 forms an image on a pointof the primary image sensor 3, the data concerning this point of thescene 1 is transmitted simultaneously by the primary image sensor 3 andthe second image data source 4.

The second image data source 4 comprises either a secondary image sensorsensing the same image as the primary image sensor 3 but with adifferent sensitivity (FIG. 3) or an image store in which image datafrom the primary image sensor 3 is stored (FIG. 2).

In the latter case the image data leaving the image store on theelectrical connection 8 may represent an image viewed previously by theprimary image sensor 3 with a different sensitivity to the imagesimultaneously leaving the primary image sensor 3, an image obtained byprocessing an image previously sent by the primary image sensor, or animage obtained from a plurality of views taken previously by the primaryimage sensor 3, combined or added together.

The data from the primary image sensor 3 and the second image datasource 4 generally represents images of the scene 1 viewed withdifferent sensitivities and so representing different ranges ofbrightness of points of the scene 1.

The image processing circuit 5 combines the data from the primary imagesensor 3 on the electrical connection 7 with the data from the secondimage data source 4 on the electrical connection 8 so that datarepresenting the scene 1 leaving the image processing circuit via theelectrical connection 5 has a ratio of the greatest brightness to theleast brightness of the points of the scene 1 represented by this imagedata greater than the dynamic range of the primary image sensor.

The image processing circuit 5 combines image data from different rangesof brightness values of the image 1 and uses each of them in full.

The combination may be a simple addition. Other modes of operation ofthe image processing circuit 5 are described later with reference toFIGS. 1 and 2.

A device in accordance with the invention can therefore be used toincrease the dynamic range of a camera whose casing B comprises theprimary image sensor 3 and the lens 2 forming am image 6 of the scene 1on the primary image sensor 3. To this end it comprises a second imagedata source 4, the data coming from at least one image of said scene 1and an image processing electronic circuit 5 combining the data from theprimary image sensor 3 and the data from the second data source 4 andsupplying image data representing the scene 1 with a ratio of thebrightest to least bright areas of the scene 1 represented by this imagedata greater than the dynamic range of the primary image sensor 3.

The conversion table 12 effects an application operation (in themathematical sense of the term) on the two sets of signal values, onereaching it on the connection 9 and the other leaving on the connection14. To this end the conversion table comprises, for example, a store towhich the address bus is connected at the connection 9 and the data busis connected at the connection 14. The store is loaded beforehand by adevice that is not shown. It may be a dynamic or static random accessmemory or a read only memory, possibly programmable. The advantage ofthe conversion table is that it formats the image data leaving thedevice so that it is easier to display on a monitor.

The flat screen 13 controlled by the circuit 10 to control thesensitivity of the primary image sensor 3 via the electrical connection18 has a transmittance which is controlled electrically over a singlearea of optical operation. To this end the flat screen 13 may comprise aflat liquid crystal screen of the nematic or ferro-electric type, forexample, between two polarizers.

The flat screen 13 varies the sensitivity of the primary image sensor 3as its transmittance varies.

The matrix flat screen 11 has an array of points at which thetransmittance is independently controlled by electrical means. To thisend the matrix flat screen 11 may comprise a liquid crystal screen, forexample, as used in some pocket TVs. An electrical signal controllingthe transmittance of each point of the matrix flat screen 11 is sentover the electrical connection 15 by the circuit 10 controlling thesensitivity of the primary image sensor 3. The transmittance of eachpoint of the matrix flat screen 11 controls the sensitivity of thepoints of the primary image sensor 3 which is optically on its outputside.

Depending on data transmitted to it on the electrical connection 15 bythe image processing circuit 5 the circuit 10 controlling thesensitivity of the primary image sensor 3 controls the transparency ofeach point of the matrix flat screen 11 according to a decreasingfunction of the brightness received by the points of the primary imagesensor 3 which are optically on the output side of said point of thematrix flat screen 11.

As a result the sensitivity of the points of the primary image sensor 3is a decreasing function of the brightness of the light impinging on it.

Finally, the opto-electronic operation of each point of the primaryimage sensor 3 is controlled by the control circuit 10.

The control circuit 10 controls the duration of this opto-electronicoperation by controlling the electronic shutter 47 of the primary imagesensor 3. To this end it sends over the electrical connection 17 eithera pulse with a duration equal to that of this opto-electronic operationor a code representing this duration, depending on which type of primaryimage sensor 3 is used. The variation of the duration of opto-electronicoperation of the primary image sensor 3 controlled by the electronicshutter 47 represents a variation of the sensitivity of the primaryimage sensor 3.

The circuit 10 also controls the control voltage of each point of theprimary image sensor 3. Finally, it controls independently the durationof opto-electronic operation of each point of the primary image sensor3. For the latter two functions the primary image sensor 3 must beadapted accordingly and matrix connections to each point of the primaryimage sensor 3 must be electrically accessible via the control circuit10.

Note that, in accordance with the invention, the potential areas orgrids of each point of the primary image sensor 3 may be controlledindividually so that the opto-electronic generation of electrical chargeat each photosensitive point is a decreasing function of the brightnessof the light reaching this point. The grid potentials are controlled bythe sensitivity control means 10 in the same way as the matrix flatscreen 11 is controlled (matrix control).

In a first embodiment the primary image sensor 3 views images withalternating high and low sensitivity under the control of thesensitivity control circuit 10. The second image data source 4 comprisesan image store which stores image data and outputs it as explained aboveand the image processing electronic circuit 5 combines the data leavingthe store and the primary image sensor 3.

In a second embodiment the primary image sensor 3 views images withconstant sensitivity, the second image data source 4 comprises a storein which the data leaving the image processing circuit 5 is stored andthe image processing circuit 5 produces a weighted mean of the imagedata from the primary image sensor 3 and the store with a weightingcoefficient varying according to the luminous intensities represented bythe image data and producing an image at the same frequency as theprimary image sensor 3. In this second mode of operation the data storedin the image store represents a sensitivity and a dynamic range greaterthan the data from the primary image sensor 3.

In a third operating mode the primary image sensor 3 operates at a highfrequency, the second image data source 4 comprises a store in whichdata leaving the image processing circuit at the same high frequency isstored and the image processing circuit combines in the store datacoming successively from the primary image sensor 3 and data stored inthe store and reading the store at a lower frequency that the operatingfrequency of the primary image sensor and so producing output image dataat this lower frequency.

In a fourth operating mode the second image data source 4 comprises anarray of stores on the primary image sensor component 3 beside eachpoint of its photosensitive surface and the image processing circuit 5controls the transfer of image data from each point of thephotosensitive surface to the juxtaposed store according to the incidentbrightness at said point and then reads the store to produce the outputscene image data. One example of this fourth mode of operation isdescribed with reference to FIG. 4.

In a fifth mode of operation the second image data source comprises astore holding flat screen 11 status data and/or data reflecting theeffect of each point of the matrix flat screen 11 and the imageprocessing circuit calculates for each point of the primary image sensor3 the transparency of the optical system on its input side and thebrightness of the area of the scene 1 from which light reaches saidpoint of the primary image sensor 3.

These five operating modes are described with reference to FIG. 2 whichshows a preferred embodiment.

In another embodiment described with reference to FIG. 3 two imagesensors simultaneously sense the same image with different sensitivitiesand supply image data to the image processing circuit which combinesthis data.

Various optical or electronic image data processing systems can be usedin these two embodiments, in particular to control the sensitivity ofthe primary image sensor 3 over all of its surface or by areas using thematrix flat screen 11 and to convert a monochrome camera into ahigh-resolution color camera.

FIG. 2 shows a scene 1 and on an optical axis A and in a camera body Bshown in part a trichrome flat screen 25, a flat screen 13, a lens 2forming an image 6 of the scene 1 on an electronic primary image sensor3, an electrically operated diaphragm 46 and a matrix flat screen 11disposed optically on the input side of the primary image sensor 3.

The primary image sensor comprises an electronic shutter 47.

Off the optical axis and in the camera casing B are shown a second imagedata source 4 comprising an image store 19, a matrix flat screen 11status store 26, a geometrical data store 27, an image processingelectronic circuit 5 having two electrical connections 7A and 7B to theprimary image sensor 3, eight electrical connections 8A, 8B, 8C, 8D, 8E,8F, 8G and 8H to the second image data source 4, the electricalconnections 8A, 8B and 8G being connected to the image store 19, theelectrical connections 8C, 8D and 8E being connected to the matrix flatscreen 11 status store 26, the electrical connections 8F and 8G beingconnected to the geometrical data store 27 and an output electricalconnection 28 connected indirectly to a conversion table 12 having anoutput electrical connection 14. A circuit 10 controlling thesensitivity of the primary image sensor 3 has its inputs connected by anelectrical connection 16 to the image processing circuit 5 and by anelectrical connection 32 to the status store 26 and has its outputsconnected by an electrical connection 17 to the primary image sensor 3,by an electrical connection 18 to the flat screen 13 and by anelectrical connection 15 to the matrix flat screen 11 and to theelectrical diaphragm 46. Outputs of the status store 26 are respectivelyconnected to an input of the geometrical data store 27 and to an inputof the control circuit 10.

The conversion table 12 has a data input connection 9 and has its outputelectrically connected to a digital-analog converter 29 having an outputconnection 30.

The image processing circuit 5 comprises a start of image detector 31connected to the output of the primary image sensor 3 and a sequencer 33controlling the grid potential having its input connected to the startof image detector 31 and its output connected to the electricalconnection 7B of the primary image sensor 3. The image processingcircuit 5 comprises an analog function circuit 34, an analog-digitalconverter 35, a multiplier 36 and an adder 37 connected in succession.The adder 37 has its output connected to the connection 7B and its inputconnected to a switch 40 whose second input is connected to theconnection 8A and whose single output is connected to the connection 9of the conversion table 12. A switch 39 connects to the adder 37 one ofits two inputs connected to a conversion table 38 and to the connection8A. The conversion table 38 has its input connected to the output of theswitch 40. Contrast measuring means 41 have their input connected to theconnection 14 and the output connected to the conversion table 12 and tothe circuit 10 controlling the sensitivity of the primary image sensor3. A blooming detector 48 has its inputs connected to the connections 8Aand 8B and its output connected to the control circuit 10. A variablethreshold circuit 42 has its input connected to the output of theprimary image sensor 3 and to the matrix flat screen 11 status store 26and its output connected to said status store 26. A computer 43 has itsinputs connected to the status store 26 and to the geometrical datamemory 27 and its output connected to the multiplier 36. A divider bytwo 44 has its input connected to the output connection 7A of theprimary image sensor 3 and its output connected to the control circuit10, to position control means for the switches 39 and 40 and to adivider by three 45 connected to the trichrome flat screen 25.

The electrical connections may comprise one or more electrical lines inorder to transmit data serially or in parallel.

A circuit D comprises the switch 40, the conversion table 12 and thedigital-analog converter 29. A circuit C comprises the circuits ofcircuit D and the analog function circuit 34, the analog-digitalconverter 35, the multiplier 36, the adder 37, the switch 39 and theconversion table 38.

The electronic primary image sensor 3 is of a known type, in particulara type known in the manufacture of video cameras. It senses correctlyand simultaneously a limited range of brightness values reaching it in agiven spectral band. Its sensitivity defines the lowest value of thisrange and depends on the transparency of the optical components whichseparate it from the scene 1. This parameter is also referred to as theaperture. Its dynamic range is defined as the ratio of the greatest andleast brightness values that it views simultaneously and correctly.Between these two extreme values it outputs image data representing thebrightness. The lowest value is equal to the sensitivity. Above thehighest value the sensor is saturated and the image data is constant. Inthe following description the term "response range" refers to the rangeof brightness values of areas of the scene 1 for which the values ofbrightness impinging on the primary image sensor 3 are between these twoextreme values.

The sensor comprises a photosensitive surface and a circuit controllingthe operation of each point of the photosensitive surface.

On the electrical connection 7A the primary image sensor 3 supplies datarepresenting the brightness received at each point of its photosensitivesurface from an area of the scene 1 and forming an image 6 on thephotosensitive surface. This image data is supplied sequentially, inother words it represents the brightness impinging on each point of theprimary image sensor 3 in succession.

The image store 19 included in the second image data source 4 supplieson the electrical connection 8A and simultaneously with the primaryimage sensor 3 image data representing the scene 1 viewed through thelens 2. This means that for each point of the scene 1 for which the lens2 forms an image at a point of the primary image sensor 3 the dataconcerning this point of the scene 1 is sent simultaneously by theprimary image sensor 3 and the second image data source 4.

The image store 19 stores image data received over the connection 8Bfrom the primary image sensor 3 following processing by the imageprocessing circuit 5. The function of the analog function circuit 34 isto amplify the weaker signals leaving the primary image sensor 3 withoutamplifying the stronger signals. The analog function circuit 34implements a logarithmic function, for example, retaining the extremevalues of the image data leaving the primary image sensor 3. Theanalog-digital converter 35 converts into the form of parallel digitaldata the signal leaving the analog function circuit 34. The multiplier36 multiplies the numerical values produced by the analog-digitalconverter 35 by numerical values produced by the computer 43. How thecomputer 43 supplies numerical values representing the transparency ofthe matrix flat screen 11 for each point of the primary image sensor 3is described later. The adder 37 adds the numerical values produced bythe multiplier 36 and the numerical values produced by the switch 39.

The divider by two 44 advises the control circuit 10 of the parity ofthe number of the image transmitted by the primary image sensor 3 andswitches the switches 39 and 40 in parallel.

For the sake of clarification, the operating modes described hereinafterexplain the function of each of the components described with referenceto FIG. 2.

In a first operating mode the circuit 10 controlling the sensitivity ofthe primary image sensor alternates the mean sensitivity of the primaryimage sensor 3 between two values supplied to it by the contrastmeasuring means 41, by controlling the flat screen 13, the electricallyoperated diaphragm 46 and the electronic shutter 47 of the primary imagesensor 43.

While the control circuit 10 selects the higher sensitivity of theprimary image sensor 3 the switch 39 connects the output 8A of the imagestore 19 to the input of the adder 37 and the switch 40 connects theadder 47 to the conversion table 12. The switches 39 and 40 are in theopposite position while the control circuit 10 selects the lowersensitivity of the primary image sensor 3. In this way the data at theoutputs 14 and 30 of the device represents the sum of the data from twodifferent response ranges of the primary image sensor 3. The dynamicrange of the camera is then the product of the dynamic range of theprimary image sensor and the ratio of the lowest sensitivity, expressedin lux, to the highest sensitivity.

The control circuit 10 controls the duration of opto-electronicoperation by controlling the electronic shutter 47 of the primary imagesensor 3. It sends over the electrical connection 17 either a pulse ofduration equal to that of opto-electronic operation or a coderepresenting this duration, depending on the type of primary imagesensor 3 used.

Note that the number of different sensitivities selectable by thecontrol circuit 10 may be more than two, in which case the switches 39and 40 have the same number of positions and the divider 44 divides bythis number.

In a second mode of operation the circuit 10 controlling the sensitivityof the primary image sensor 3 does not alternate the sensitivity of saidsensor. The switches 39 and 40 are locked in such a way that theconversion table 38 is connected to the adder 37, the setting of theswitch 40 being immaterial. The conversion table 38 defines a functionwhich increases and then decreases according to the numerical values "x"arriving at its input. For example, this function is of the form

    ((nth root of x) less x) divided by maximum x.

Accumulation therefore occurs in the image store 19, in inverseproportion to the value x. In this way the sensitivity and dynamic rangeof the camera are increased and the remanence effects are restricted tothe lowest values for which the effects are less perceptible. Theincreased dynamic range results from the increased signal to noise ratiodue to the high accumulation for low values of x for which the noise isgreater. For the forms of function mentioned above the response of thecamera is of the nth root of x form.

In a third mode of operation the device operates as in the second modeof operation except that the switch 40 is in a fixed setting and theconversion table 12 is connected at all times to the image store 19. Theimage sensor operates at a high frequency and the conversion table 12reads the image store 19 at a low frequency. In this way the increaseddynamic range of the camera is similar to that explained in the secondmode of operation but the remanence is attenuated.

In a fourth mode of operation the device operates as in the third modeof operation except that the setting of the switch 39 changes and thisswitch does not connect either of its inputs to its output during aperiod in which the primary image sensor 3 produces an image after eachperiod of reading the image store 19 by the conversion table 12. In thisway there is no remanence at the output of the device because the datain the image store 19 is renewed on each read.

Fifth and sixth modes of operation use control of the matrix flat screen11 to vary the sensitivity of the primary image sensor 3 matrix fashion.

The matrix flat screen 11 has an array of points whose transmittance iscontrolled individually by electrical means. To this end the matrix flatscreen 11 may comprise a liquid crystal screen, for example, as used insome pocket TVs. An electrical signal controlling the transmittance ofeach point on the matrix flat screen 11 is sent over the electricalconnection 12 by the circuit 10 controlling the sensitivity of theprimary image sensor 3. This signal is supplied to the control circuit10 by the status store as explained hereinafter. The transmittance ofeach point of the matrix flat screen 11 controls the sensitivity of thepoints of the primary image sensor 3 which are optically on its outputside.

Depending on the data transmitted to it over the electrical connection16 by the image processing circuit 5, the circuit 10 controlling thesensitivity of the primary image sensor 3 controls the transparency ofeach point of the matrix flat screen 11 according to a decreasingfunction of the brightness received by the points of the primary imagesensor 3 which are optically on the output side of said point of thematrix flat screen 11.

In this way the sensitivity of the points of the primary image sensor 3is a decreasing function of the brightness of the light impinging on it.

For any point of the matrix flat screen 11 the geometrical data store 27holds values defining its optical effect at each point of the primaryimage sensor 3 which is optically on its output side. These valuesdepend on the distance between the matrix flat screen 11 and the primaryimage sensor 3, on the position and the aperture of the electricaldiaphragm 46 and on the darkened state of the points of the matrix flatscreen 11 surrounding said any point. This effect is the result of theprojection of points of the matrix flat screen 11 onto thephotosensitive surface of the primary image sensor 3 by the light raysfrom the scene 1 via the electrical diaphragm 46. The electricaldiaphragm 46 being connected to an input of the geometrical data store27, together with the matrix flat screen 11 status store 26, thegeometrical data store 27 can supply the optical effect values to thecomputer 43. This computes the reciprocal of the optical attenuationfactor of light rays impinging on each point of the primary image sensor3. This reciprocal is multiplied by the value of brightness at saidpoint by the multiplier 36, the output values of the multiplierrepresenting a constant overall sensitivity of the photosensitivesurface of the primary image sensor 3. Because it depends on the actualscene 1, the control of the matrix flat screen 11 is stable. Thisrepresents the fifth mode of operation.

In the sixth mode of operation of this device in accordance with thepresent invention the matrix flat screen 11 status store 26 preventsimage oscillations due to the closed loop control. To this end thevariable threshold circuit 42 compares for each photosensitive point ofthe primary image sensor 3 the brightness value impinging on this point,which value is supplied over the connection 7A from the primary imagesensor 3, with a threshold dependent on the transparency of the point ofthe matrix flat screen 11 which is optically on the input side of thispoint of the sensor, as indicated by the state store 26. Depending onthis transparency a threshold is determined. If the threshold isexceeded for at least one point of the primary image sensor 3 on theoutput side of said point of the matrix flat screen 11 the variablethreshold circuit 42 commands maximum darkening at this point of thematrix flat screen 11 by sending data to the state store 26. If thethreshold is not exceeded for any point of the primary image sensor 3which is optically on the output side of said point of the matrix flatscreen 11 the variable threshold circuit commands the maximaltransmittance at this point of the matrix flat screen 11 by sendingdifferent data to the state store 26. In this way, for a fixed scene 1,the flat screen 11 commands are constant, the maximum thresholdvariation representing the maximum contrast of the matrix flat screen11. No oscillation of the status of points of the matrix flat screen 11or of the image data leaving the primary image sensor 3 is possible.

Note that for each point of the matrix flat screen the ratio of therespective optical attenuation on the point of the primary image sensor3 which is optically on its output side to the minimal opticalattenuation is always less than the ratio of the threshold used to theminimal threshold. It is this geometrical optical effect of projectionby the light rays passing through the electrical diaphragm 46 of pointsof the matrix flat screen 11 onto the primary image sensor 3 whichrenders control stable.

Note also that this mode of operation is described for a flat screenhaving only two states, one transparent and the other opaque, at eachpoint. It will be easy for the man skilled in the art to adapt thisdisclosure to a flat screen having intermediate levels of transparency,thresholds representing these levels of transparency being then usedsubject to the stability of control rule explained above.

For the fifth and sixth modes of operation note that the data stored inthe state store 36 represents at least one image of the scene 1 viewedthrough the lens 2. The image processing circuit 5 combines this datawith data from the primary image sensor 3 and outputs image data of thescene 1 representing a dynamic range greater than that of the primaryimage sensor 3.

A seventh operating mode uses the start of image detector 31 and thegrid potential control sequencer 33.

The start of image detector 31 advises the grid potential controlsequencer 33 of the start of an image leaving the primary image sensor 3in the form of image data.

The sequencer 33 then sends potential signals to the grids of theprimary image sensor 3 in such a way that a partial transfer ofelectrical charge occurs from its surface to its output 7A as explainedlater, in an example referring to a CCD primary image sensor structure.In this example, for a primary image sensor 3 comprising four potentialareas side by side for each point of the photosensitive area, saidpotential areas have a potential commanded by the sequencer 33, fourgrids each defining a potential for all the identical areas of thesepoints are accessible by the sequencer 33. Thus all the photosensitivepoints of the primary image sensor 3 have simultaneously the sameelectrical potential configuration on their four potential areas. By wayof explanation, there will be described the operation of a primary imagesensor 3 controlled by the sequencer 33 and having four grids of whichthe first represents the photosensitive areas of each photosensitivepoint, the next two represent the charge transfer areas and the lastrepresents a potential barrier or drain preventing electrical connectionbetween the areas of two adjacent points. The last three grids thereforerepresent areas of points of the primary image sensor 3 which are notphotosensitive and do not have any opto-electronic function.

Throughout the operation as described hereinafter, the fourth grid (i.e.the potential barrier or drain) remains in a state such that theelectrons cannot cross it.

In a first time period the sequencer 33 commands opto-electronicoperation of long duration at the photsensitive areas, i.e. the firstgrid. At this time the second and third grids do not contain anyelectrical charge. In a second time period the sequencer commands thesame potential at the first three grids. In this way the electricalcharges initially in the first area are partially transferred into thesecond and third areas. In a third time period the potential of thesecond grid is modified so that the electrical charges can no longerpass from the first to the third area. In a fourth time period thepotential of the first grid is modified so that the charges retained bythe first area are eliminated or transferred to a substrate or a drain.At this time only the third area retains electrical charges, the sum ofthese electrical charges being a fraction of the sum of the electricalcharges generated in the first time period in the first area. Saturationof the third area is therefore impossible. In a fifth time period thepotential of the first grid is changed to restart opto-electronicoperation of the first area during a period shorter than the first timeperiod. In a sixth time period the electrical charges in the first areaare transferred in their entirety into the third area via the secondarea in the usual way. Finally, in a seventh time period the charges aretransferred from the third area to an image data output of the primaryimage sensor 3 in the usual manner. In this way these charges representsome of the charges resulting from the first period of opto-electronicoperation and charges resulting from the fifth period of opto-electronicoperation.

This seventh mode of operation of the device makes it possible toimplement the same operations as in the first mode of operation in astore in the primary image sensor 3, the store being made up of thethird photosensitive point areas of said primary image sensor 3. Inother words, the image data leaving said store on the primary imagesensor 3 includes data representing two views with differentsensitivities of the primary image sensor 3. This data represents adynamic range greater than that of the primary image sensor 3.

Note that the second and third time periods described above can berespectively replaced by the eighth and ninth time periods describedbelow. The eighth time period is identical to the second time periodexcept that the third grid is at a potential which does not allowtransfer of electrical charges within it. The ninth time period isidentical to the third time period, the potentials of the second andthird areas being progressively interchanged. The result in respect ofthe positions of the electrical charges in the fourth time period is thesame as previously.

Other grid or potential control sequences sent to the primary imagesensor are within the scope of the invention, especially if the numberof areas per point of the primary image sensor or the number of gridsused is different than that used hereinabove. Another example of gridpotential control sequence effected by the sequencer 33 is shown in FIG.4. For a better understanding of this seventh mode of operationreference may usefully be had to works describing the physics ofsemiconductor active components and in particular the work on thissubject by P. Leturcq and G. Rey published in the French language byeditions Dunod, Paris, and in particular pages 145 through 150 which arehereby incorporated in the present description.

In addition to the seven modes of operation described above and whichare combined to add their dynamic range enhancement effects, theinvention proposes various systems of enhancing image data quality,especially in the case of blooming or smearing.

The effect of smearing in images from the primary image sensor 3 isreduced by darkening the flat screen 13 when the primary image sensor 3is not sensitive, in other words when opto-electronic operation is notrequested, or when the electronic shutter is closed. In a device inaccordance with the present invention as shown in FIG. 2 the circuit 10controlling the sensitivity of the primary image sensor 3 controls thedarkening of the flat screen 13 by means of a clock (not shown).

The contrast measuring means 41 measures the variations in the numericalvalues leaving the device. It can measure the quadratic difference ascompared with the mean numerical value. It can also detect the extremenumerical values. The contrast measurement is supplied to thesensitivity control circuit 10 and to the conversion table 12. Thesensitivity control circuit 10 uses this data to control variations inthe sensitivity of the primary image sensor 3 accordingly. Theconversion table comprises a plurality of transfer functions suited todifferent values of contrast for optimizing the display of image dataleaving the device.

The blooming detector 48 compares the values entering and leaving theimage store 19. These images represent different sensitivities. Toprevent blooming occurring in the most sensitive image, the bloomingdetector 48 verifies that for the saturation value of the data from animage viewed with the highest sensitivity there is a non-null value ofdata from an image viewed with the lowest sensitivity. The datarepresenting these two images is present on the connections 8A and 8B inthe first mode of operation of the device. The detection of blooming isreported to the sensitivity control circuit 10 which commands a smallerdifference in sensitivity between successive images by means of the flatscreen 13, the electric diaphragm 46 and the electronic shutter 47.

The conversion table 12 can be used to enhance the image leaving thedevice. In particular, conversion functions which increase and thendecrease create a polarization effect enhancing the perception of theimage.

With a monochrome primary image sensor 3 three circuits D must beincorporated into the device, each representing one of the primarycolors. The electrical connection 8H connecting the divider by three 45and the image store 19 selects the memory plane of the image store 19representing the color transmitted by the flat trichrome screen 25 asdescribed hereinafter. The flat trichrome screen 25 which is controlledby the circuit 10 controlling the sensitivity of the primary imagesensor 3 has a transmittance controlled electrically in a singlecooperating area for each of the primary colors. To this end the flattrichrome screen 25 may incorporate three flat liquid crystal screens,for example nematic or ferroelectric screens, and three dichroicpolarizers between which are disposed the liquid crystal flat screensand a conventional polarizer. The dichroic polarizers filter one coloron one polarization axis and are transparent on a polarization axisperpendicular to the first. The conventional polarizer is transparent onone polarization axis and opaque on the polarization axis perpendicularto the first. The alternating disposition along the optical path of ared-transparent dichroic polarizer, a liquid crystal screen, agreen-transparent dichroic polarizer, a liquid crystal screen, ablue-transparent dichroic polarizer, a liquid crystal screen and aconventional polarizer selects the color transmitted by the flat shutter13.

The transmittance of the flat trichrome screen 25 varies the sensitivityof the primary image sensor 3 for each of the primary colors.

With a color primary image sensor 3 three circuits C must beincorporated into the device each representing one primary color. Aswitch (not shown) at the input of each circuit C selects the datarespective to each color. The flat trichrome screen and the divider bythree 45 are not required.

Opto-electronic operation of each point of the primary image sensor 3 iscontrolled by the control circuit 10.

The control circuit 10 controls the operating voltage of each point ofthe primary image sensor 3. It controls independently the duration ofopto-electronic operation of each point of the primary image sensor 3.The primary image sensor 3 must be adapted for these functions andmatrix connections with each of the points of the primary image sensor 3must be electrically accessible to the control circuit 10.

It is clear that a device in accordance with the present invention maycomprise fewer functions than all those described with reference toFIGS. 1 and 2 whilst achieving in accordance with the invention anincrease in the dynamic range of the primary image sensor.

In each operating mode the image data held by the second image datasource represents an image with a range of brightness variationdifferent than that of the primary image sensor 3, for which this rangeof variation is called the dynamic range.

FIG. 3 shows a scene 1 and on an optical axis A and in a camera casing Bwhich is partially shown a flat screen 13, a lens 2 forming an image 6of the scene 1 on a primary image sensor 3, light splitting means 21 anda matrix flat screen 11 disposed optically on the input side of theprimary image sensor 3. Off the optical axis and in the camera casing Bare shown a second image data source 4 comprising a secondary imagesensor 23 on which the lens 2 forms an image 24 of the scene 1 similarto the image 6, an image processing electronic circuit 5 having anelectrical connection 7 to the primary image sensor 3, an electricalconnection 8 to the second image data source 4 and an output electricalconnection 9 connected to a conversion table 12 having an outputconnection 14. A circuit 10 controlling the sensitivity of the primaryimage sensor 3 is connected by an electrical connection 16 to the imageprocessing circuit 5, by an electrical connection 17 to the primaryimage sensor 3, by an electrical connection 18 to the flat screen 13 andby an electrical connector 15 to the matrix flat screen 11.

The components and functions of this second embodiment of the device areidentical to those described with reference to FIGS. 1 and 2 except forthe second image source 4 and the light splitting means 21.

The function of the light splitting means 21 is to split light raysimpinging on it into two light rays of which one propagates towards theprimary image sensor 3 and the other towards the secondary image sensor23. To this end it may comprise partially reflecting mirrors,transparent plates, optionally with surface treatment, optical prismssuch as those used in cameras with three image sensors each sensing onecolor, or an image transfer lens.

The second image data source 4 comprises a store identical to thatdescribed with reference to FIG. 2. It also comprises a secondary imagesensor 23 which senses an image 24 similar to the image 6 with asensitivity different than that of the primary image sensor 3. To thisend the transparency or the aperture of the optical system in front ofthe secondary image sensor 23 is different than that of the opticalsystem in front of the primary image sensor 3.

The image sensors 3 and 23 therefore provide representative data fordifferent ranges of values of brightness of points of the scene 1. Theimage processing circuit 5 combines this data to provide on anelectrical output connection 9 data representing the combination ofthese two ranges of brightness values of points of the scene 1.

FIG. 4 shows at the top the grids of the charge-coupled devices of anelectronic image sensor and, lower down, vertically aligned with thelatter, grid potential states at consecutive times, these potentialsbeing imposed by the grid potential sequencer shown in FIG. 2.

At the top are shown grids G1, G2, G3, G4 on a substrate 50, grids G5,G6, G7 on a substrate 51 and grids G8 and G9 on a substrate 52. Thesubstrates 50, 51 and 52 respectively represent image sensors with four,three and two grids per photosensitive point of the photosensitivesurface.

Masks 53 shade the grids G2, G3, G4, G6, G7 and G9 which are intended totransfer charge created opto-electronically to the grids G1, G5 and G8.

The first curved line L1 represents the potential during viewing, i.e.during formation of charges on the grids G1, G5 and G8. The subsequentlines show, for successive times from top to bottom, the transfer ofcharge between the grids. Points on these curves above the axesrepresent positive potentials and below the axes represent negativepotentials. The positive potential points constitute a barrier forholes, these being the charge carries transferred in the charge-coupleddevices. The negative potential points represent potential wells wherethe hole charges are retained. The charge positions are represented bythe symbol "+".

In line L1, grids G1, G5 and G8 have a negative potential and the othergrids have a null potential. At this time the positive electricalcharges are generated and retained by the grids G1, G5 and G8. In lineL2, the grids G1, G2, G3, G5, G6 have a negative potential, the othergrids having a null potential except for the grid G8 which has a verylow negative potential. At this time the charges from grids G1 and G2are respectively distributed under the grids G2 and G3 on the one handand the grids G6 and G7 on the other hand. The charges generated underthe grid G8 are partially dissipated into the grid G8. In line L3 gridsG1, G3, G5, G7 and G9 are at negative potentials, the other grids beingat a null potential. At this time only some of the charges initiallygenerated under the grids G1, G5 and G8 are retained under the grids G3,G7 and G9. In line L4 grids G3, G7 and G9 are at negative potentials,the other grids being at a null potential except for grid G8 which is atan intermediate negative potential. The charges present under the gridsG1 and G5 are absorbed by the grids G1 and G5. New charges are generatedunder the grid G8, representing a second view. In line L5 grids G1, G3,G5, G7 and G9 are at negative potentials, the other grids being at anull potential. At this time new charges have been generated under thegrids G1 and G5 representing a second view. Charges created during thesecond view (line L4) under grid G8 are transferred under grid G9. Afterline L5 the cycle of operation of grids G8 and G9 is completed and thepotential state of the grids is not shown. In line L6 grids G1, G2, G3,G4, G5 and G6 are at a negative potential, grid G4 being at a nullpotential. In line L7 grids G2, G3, G6 and G7 are at negativepotentials, grids G1, G4 and G5 being at null potentials. In line L8grids G3 and G7 are at negative potentials, grids G1, G2, G4, G5 and G6being at null potentials. The last three lines represent the transfer ofcharges generated during the second view (line L5) under grids G1 and G5under grids G3 and G7.

At the end of this sequence grids G3, G7 and G9 retain charges whosenumber is the sum of the charges created during the second view withsome of the charges created during the first view (line L1).

The second view is preferably of shorter duration or at lowersensitivity than the first. In this way saturation cannot occur.

Grids G3, G7 and G9 form an array of stores disposed in the primaryimage sensor constituting in accordance with the present invention asecond image data source.

Other sequences, in particular adapted to other types of CCD imagesensors, may be used by the man skilled in the art within the scope ofthe invention. Note that in accordance with the invention the potentialareas of grids of each point of the primary image sensor can becontrolled individually so that the opto-electronic generation ofelectrical charges at each photosensitive point is a decreasing functionof the brightness of light impinging on this point. The potential of thegrids is controlled by the sensitivity control means 10 in the same wayas the matrix flat screen 11 is controlled (matrix control).

A device in accordance with the present invention may be implemented foraddition to any previously existing electronic camera or incorporatedinto the casing of a camera.

Starting with any monochrome camera, it can increase the dynamic range,increase the sensitivity and convert the camera into a high-definitioncolor camera.

The main applications of a device in accordance with the presentinvention are shooting under uncontrolled light conditions outdoors, inendoscopy and in welding robots, for example.

I claim:
 1. Device for increasing the dynamic range of an electronicimage sensor comprising photosensitive elements for supplying successivesignals representing images formed successively at the photosensitiveelements, and sensitivity control means for varying the lowest level ofbrightness of images successively sensed by the electronic image sensorfor each image, an image store having an input electrically connected tosaid image sensor, said image store being adapted to store the imagedata received from said image sensor and send a signal previouslystored, image data simultaneously output by said image sensor and saidimage store corresponding to a same one of said photosensitive elementshaving different lowest levels of brightness, and electronic imageprocessing circuit means for combining image signals received directlyfrom said image sensor and image signals received via said image storefor supplying at an output connector an image signal having a greaterdynamic range than said image sensor.
 2. Device according to claim 1,further comprising a flat screen optically in front of the image sensorand having an electronically controlled transmittance area, thesensitivity control means controlling the transmittance of the flatscreen so that the transmittance of the flat screen is alternately highand low during viewing by the electronic image sensor.
 3. Deviceaccording to claim 2, wherein the sensitivity control means senses acontrol signal representing the low transmittance of the flat screenbetween successive views by the image sensor.
 4. Device according toclaim 1, wherein the sensitivity control means controls the sensitivityof the elements of the image sensor matrix fashion and varies thesensitivity independently.
 5. Device according to claim 4, wherein thesensitivity control means independently controls opto-electronicoperation duration of each of the photosensitive elements of the imagesensor.
 6. Device according to claim 4, wherein the sensitivity controlmeans independently controls conversion of photons into electrons ateach of the photosensitive elements of the image sensor.
 7. Deviceaccording to claim 4, further comprising a passive matrix flat screen,the sensitivity control means controlling the transparency of the matrixflat screen, said matrix flat screen being disposed optically in frontof the image sensor, the transparency of each of the elements being adecreasing function of brightness represented by the image signal at theoutput of the image processing circuit and the respective brightness atthe photosensitive elements of the image sensor disposed optically aftersaid element of the matrix flat screen.
 8. Device according to claim 7,further comprising a second image store holding data on the transparencyof each of the elements of the matrix flat screen, said second storehaving an output connected to the input of the image processing circuit.9. Device according to claim 4, further comprising a geometrical datastore holding at least one sensitivity indication for each of theelements of the image sensor.
 10. Device according to claim 1, furthercomprising blooming detector means for comparing point by point datarepresenting two images viewed successively by the electronic imagesensor with different sensitivities, the output of the blooming detectormeans being connected to an input of the sensitivity control means. 11.Device according to claim 10, wherein the sensitivity control meansreduces the sensitivity differential between successive images when theoutput from the blooming detector means indicates blooming.
 12. Deviceaccording to claim 1 wherein the sensitivity control means is adapted tovary the lowest level of brightness of images sensed successively bysaid image sensor.
 13. Device according to claim 1, wherein the imagestore is incorporated into the image sensor and comprisesnon-photosensitive areas adjacent photosensitive areas of the imagesensor and each non-photosensitive area is adapted to retain a sum ofpart of electrical charges generated during a first period ofopto-electrical operation and another part of electrical chargesgenerated during a second period of opto-electrical operation. 14.Device according to claim 13, wherein the image sensor is a CCD sensorcontrolled by grids and said device further comprises a grid potentialsequencer for varying the potential of said grids for each of thephotosensitive elements of the electronic image sensor so that differentsensitivities are used for two successive views and the electricalcharges generated during the first of the views are partially retainedin the non-photosensitive area and partially eliminated and saidelectrical charges retained in the first of the views are added toelectrical charges generated during the second of the views.
 15. Deviceaccording to claim 1, wherein the image store simultaneously stores thesignal emitted from the image sensor and sends a signal previouslystored, said image store retaining the image data of the image signalfor a time period equal to that required by the image sensor to sendcomplete image data, the image data simultaneously output by the imagesensor and the image store relating to the same photosensitive elementof the image sensor.
 16. Device according to claim 15, wherein the inputof the image store is electrically connected to an output of the imageprocessing circuit so that the signal output stored by the image storeis the signal processed by the image processing circuit.
 17. Deviceaccording to claim 16, further comprising switch means having alternatesettings for each view from the electronic image sensor and two inputselectrically connected respectively to the output of the image store andto the output of the image processing circuit and also having an outputelectrically connected to an output connector supplying an output signalfor alternate images which is the output signal from the image store andan output signal for intervening images which is the output signal fromthe image processing circuit.
 18. Device according to claim 16, whereinthe image processing circuit includes means for reading the image storeat a frequency lower than the operating frequency of the image sensor.19. Device for increasing the dynamic range of an electronic imagesensor comprising photosensitive elements for supplying successivesignals representing images formed successively at the photosensitiveelements, and sensitivity control means for varying the sensitivity ofthe electronic image sensor for each image, wherein the improvementcomprises an image store having an input electrically connected to saidimage sensor, said image store being adapted to store the image datareceived from said image sensor and send a signal previously stored,image data simultaneously output by said image sensor and said imagestore corresponding to a same one of said photosensitive elements, andelectronic image processing circuit means for combining image signalsreceived directly from said image sensor and image signals received viasaid image store for supplying at an output connector an image signalhaving a greater dynamic range than the image sensor, and an electronicshutter coupled to said image sensor, the sensitivity control meanscontrolling said electronic shutter by alternating viewing time forsuccessive views between a short viewing time and a long viewing time.20. Device for increasing the dynamic range of an electronic imagesensor comprising photosensitive elements for supplying successivesignals representing images formed successively at the photosensitiveelements, and sensitivity control means for varying the sensitivity ofthe electronic image sensor for each image, wherein the improvementcomprises an image store having an input electrically connected to saidimage sensor, said image store being adapted to store the image datareceived from said image sensor and send a signal previously stored,image data simultaneously output by said image sensor and said imagestore corresponding to a same one of said photosensitive elements, andelectronic image processing circuit means for combining image signalsreceived directly from said image sensor and image signals received viasaid image store for supplying at an output connector an image signalhaving a greater dynamic range than said image sensor, the imageprocessing circuit means comprising a conversion circuit seriallyconnected to an adder, the conversion circuit having an inputelectrically connected to the output of said image store and an outputconnected to the image store via the adder, said conversion circuitimplementing an increasing and then decreasing transfer function. 21.Device for increasing the dynamic range of an electronic image sensorcomprising photosensitive elements and supplying successive image datarepresenting images formed successively at the photosensitive elements,wherein the improvement comprises an image store having an inputelectrically connected to the image sensor, said image store beingadapted to store image data from the image sensor and send previouslystored image data, the image data simultaneously output by the imagesensor and by the image store relating to a same one of thephotosensitive elements, a conversion circuit serially connected to anadder and together defining image data processing means, the image dataprocessing means having an input electrically connected to the output ofthe image store and an output electrically connected to the input of theimage store, the adder also being electrically connected to the imagesensor for adding image data from the image sensor and the image store,and the conversion circuit implementing an increasing and thendecreasing transfer function.